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VIRTUAL PAN/TILT CAMERA SYSTEM AND METHOD 
FOR VEHICLES 

Statement Regarding Federally Sponsored Research or Development: 
5 The subject matter of the present disclosure Th i s inventio n was 

developed in the course of work under U.S. government contract MDA972-G2-9- 
0005. The U,S, government may possess certain rights in the in v e ntion subject 
matter of the present disclosure . 



10 Background of th e Inv e nt i on 

The present disclosure Thi s invention is directed to a camera system for 
vehicles for use with autonomous navigation systems (ANSs), and more particularly 
to a virtual pan/tilt camera system and method for use with ANSs for ground 
vehicles, more specifically manned ground combat vehicles. 

15 Unmanned ground vehicles (UGVs) are valuable to the military involved in 

ground combat. The value of a UGV is increased when it is able to navigate 
autonomously as it reduces manpower that must be deployed for a given military 
operation. Modem ANSs for ground vehicles use machine vision to detect obstacles 
and navigate paths around the obstacles, Current ANSs use pan/tilt stereo cameras 

20 oh the outside of the UGV so that the ANS computer can get a good view of the 
obstacles at dose range. 

ANSs used in modem unmanned ground vehicles operate using as sources of 
information: (1 ) a digital map of the terrain where the vehicle is operating that is 
precise to approximately a meter or better, and (2) redefined information that it 

25 deduces from the pan/tilt cameras used in the system. It also uses GPS coordinates 
to establish the vehicle's location. However, before the ANS directs the vehicle to 
begin moving, it may require a closer view of the area immediately surrounding the 
vehicle. 
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The autonomous navigation of ground vehicles is particularly difficult. To 
autonomously navigate over ground, detailed knowledge is required of objects down 
to a size that the vehicle can traverse- Therefore, ground navigation requires 
continuous detailed inspection of the area immediately ahead of the vehicle. For 
5 example, there may be obstacles such as trees, brush, rubble, boulders, and other 
objects that do not appear on the digital map. From a distance, the ANS can identify 
that a particular object is present, but may not be able to tell whether it is 
surmountable, Le u whether the vehicle can travel over the object. To further identify 
and characterize such objects, the ANS causes the vehicle to navigate closer to the 

10 objects, and as it does so initiates suitable commands to the pan/tilt cameras to aim 
the cameras at the various nearby objects to gain better imagery of the objects- 
Objects that are classified as "impassable'- are added to a dynamic map. The ANS 
selects an appropriate route, if such a route is available, and initiates commands to 
direct the vehicle between or around the impassable objects, and through or over the 

15 passable ones. 

Pan/tilt cameras work well for an ANS and do not pose any excessive 
demands for most UGVs, How e v e r r prob te rosoccur - w i th th e- u se of ANSs on 
manned ground v e h i c le s (MGV s ). To enable MGVs to travel while a crew is 
resting or disabled, and to enable the vehicles to operate unmanned when facing 

20 extreme danger, it is required that the MGV be able to operate the vehicle by remote 
control and navigate autonomously, such that the MGV must include an ANS, 
Because the current state of ANS technotogy requires a pan/tilt camera, a standard 
approach is to incorporate a pan/tilt camera into each MGV. How e v e r, th e us e of a 
pan/t i lt camera w i th an MGV presents prob l ems not a ssoc ia ted with the use of 

25 such a camera w i th a UGY. 

On e such prob l em i s th a t the p a n/t il t camera i ncre a ses veh i c l e w e ight 

and power needs wh i ch -is of p a rt i cu la r conc e rn for MGV s wh e r e w e ight and 
pow e r a r e s e v e r e ly constr a in e d. Anoth e r probl e m i s th a t th e pan/t i lt mount for 
th e c a m e ra is difficu l t to h i d e from radar and infrar e d se nsors, th e r e by 

30 comprom is ing th e low - obs e rv a bl e ch a r a ct e r i st i cs n ee d e d for m i l i t a ry vehic le s 
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to surv i vo on the modern battlef i e l d. Thi s i nvent i on mitigates those prob l ems 
by elim i nating th e n ee d - fo f- th e- p a n/t il t camera without requiring the 
d e v e lopm e nt of n e w ANS t e chnology and without i ncurr i ng the co s ts and 
schedu l e de l ays assoc i ated with s uch d e ve l opm e nt 

5 

Summary of tho Inventio n 

In accordance with various embodiments of the present disclosure this 
invention the problems a ssoc i ated w i th th e us e of a pan/tilt cam e ra w i th 
pro s ont ANS technology in an MGV are e l iminated by o li mma tt ng - the pan/t il t 

10 cameras on th e v e h i cl e and repl a cing them with a plurality of video infrared 
cameras that give the vehicie crew a view of the horizon in ail directions are 
mounted on the vehicle . Each of the cameras is mounted in a fixed orientation on 
the vehicle and covers a selected angular range of lateral view, preferab l y so that 
such that in various embodiments every point on the horizon is viewed by two 

1 5 different cameras such that the cameras' fields of view overlap enabling every point 
on the horizon to be viewed even when a camera is disabled. 

Computers and software in the MGV continuously receive data from all of the 
cameras, store the data in video frame buffers, and stitch the imagery together into a 
single, seamless panoramic view, in a various preferred embodiments of the 

20 invent i on disclosure , each crew member of an MGV can select which subset of the 
view to display on a particular computer console. A display computer maps the 
selected view to a rectangular region of a video frame buffer, and data from that 
region is sent to the particular computer console each time a new video frame eomes 
into the buffer A crew member of the MGV can adjust the selected view to the right 

25 or left, up or down, or zoom in or out The computers and software map pan/tilt 
commands for individual cameras from the MGV's ANS to regions of the MGV's 
video frame buffers, and feed appropriately transiformed data from the video frame 
buffers to the ANS. Video data from two different cameras with overlapping fields of 
view are used to simulate the view of stereo cameras. 
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Thus, the present disclosure i nvention eliminates the need for pan/tilt 
cameras and the disadvantage s associated with their use on MGVs, by using a 
plurality of fixed cameras, each covering a selected angular range of lateral view with 
the imagery from the cameras stitched together to produce a single, seamless 

5 panoramic view as may be selected by a crew member With the present 

disclosure i nvont i on , the MGV does not suffer penalties to its radar and infrared 
signatures as would may occur with the use of pan/tilt cameras and their associated 
mounts. Moreover, the present disclosure i nv e nt i o n may be used with present 
ANS technology thereby eliminating the substantial costs that would be associated 

10 with developing a new ANS. The present disclosure invontion reduces weight and 
power requirements as compared to the use of pan/tilt cameras. 

Further features and advantages of the present disclosure invont i on , as well 
as the structure and operation of various embodiments of the present disclosure 
invont i on , are described in detail below with reference to the accompanying 

15 drawings. 



Brief Description of the Drawings 

Figure 1 is a schematic block diagram of a pr e f e rr e d e mbod i m e nt of the 
present disclosure i nvention in accordance with various embodiments : 
20 Figure 2 is a diagram illustrating image distortion; 

Figure 3 is a diagram illustrating a geometrical transformation in accordance 
with a pref e rred embodiment of the present disclosure invention in accordance 
with various embodiments; and 

Figure 4 is an illustration of image transformation i n accordanc e w i th a 
25 proforrod e mbod i ment of the present disclosure i nv o ntion in accordance with 
various embodiments . 



Detailed Description-ofrt ho Preferred Embod iments 

With reference to the drawings, there i s shown a proforrod are illustrated 
3G various embodiments of a virtual pan/tilt camera system 1 Q in accordance with the 
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present disclosure i nvention for use with a manned ground combat vehicle 12 as 
illustrated in Figure 3. The system generally comprises a fixed camera video data 
input 14, a reformatting network section 16, and a display system section 18. The 
fixed camera video data input 14 further comprises a plurality of video cameras 20, 
5 each of which is mounted in a fixed position on the vehicle 1 2. In this embodiment 
eight such cameras are shown, but it is to be understood that a fewer or greater 
number of cameras may be used. A suitable camera is the Hitachi HV-C20 or the 
Sony DXC 9000. Each camera outputs video data 22 representing the images 
viewed by that camera to a frame buffer 24, there being eight such frame buffers, 

10 one for each video camera. Each frame buffer stores at addresses within the buffer 
the video data it receives from its respective camera. 

An ANS 26 of the same type used for autonomous navigation of unmanned 
ground vehicles that use two pan/tilt cameras is programmed to operate as if it were 
in a system using two pan/tilt cameras. Thus, as previously explained, the ANS is a 

15 pre-existing system that performs stereo image processing, and includes appropriate 
hardware and software to autonomously navigate the vehicle as is known in the art. 
An example of such a system is an ANS incorporated in the experimental unmanned 
vehicle developed by General Dynamics Robotic Systems and the National Institute 
of Standards and Technology for the Department of Defense's Demo III program. 

20 What the ANS needs in order to perform its required functions is two video input 
streams from a pair of pan/tilt cameras that it can control. With the system 10 of the 
present disclosure invention , there are no pan/tilt cameras. Instead, the system 10 
functions to deliver two video input streams generated from fixed cameras, but which 
have been reformatted to appear to have come from two pan/tilt cameras. 

25 Thus, the ANS determines the best angle at which to point what would be two 

pan/tilt cameras to get a view of an obstacle, and outputs appropriate pan/tilt 
commands 28 to the reformatting network system 16. The system 16 comprises a 
Translate to Az-EI component 30, an Az-EI to Address Translation component 32, 
and a Video Formatting component 34. The pan/tilt commands 28 from the ANS 26 

30 are input to the Translate to Az-EI component 30 which converts the pan/tiit 
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commands into signals representing the azimuth and elevation for each pan/tilt 
camera as if such cameras were present in the system. The azimuth and elevation 
output signals 36 from the Translate to Az-EI component 30 are input to the Az-EI to 
Address Translation component 32 which generates address signals 40 which are 
5 input to each of the video frame buffers 24. The Az-EI to Address Translation 

component 32 maps the azimuth and elevation signals, as commanded by the ANS, 
to addresses in the video frame buffers 24, thereby determining which of the frame 
buffers 24 and which data stored in each buffer to access for generating each 
section of the image. 

10 Video data 50 from the specified addresses, as determined by the Az-EI to 

Address Translation component 32, in each of the frame buffers 24 are input to the 
Video Formatting component 34 which stitches together images from adjacent frame 
buffers, if needed, and reformats the images to correct the image geometry, made 
necessary because the fixed cameras 20 do not actually point in the direction 

15 specified by the pan/tilt commands 28 from the ANS 26. An image formed to the 
side of an image plane of a camera 20 is distorted compared to the same image 
formed at the center of the image plane as illustrated in Figure 2. Figure 2A 
illustrates an image, represented by arrows 60, 62, centered on a camera axis 64, 
only minimally distorted at the focal plane 66 as shown by the arrows 68, 70 of 

20 substantially equal length. Figure 2B shows the same image, represented by arrows 
60. 62, centered off the camera axis 64, distorted at the focal plane 66 as shown by 
the arrows 72, 74 of unequal length in the image plane. 

This is further illustrated and explained with reference to Figures 3A and 3B. 
Figure 3A illustrates a ground combat vehicle 12 having two pan/tilt cameras A and 

25 B, each pointing directly, or nearly directly, to an image represented by arrows 80, 
82. That is, the camera axes 84, 86 of the cameras A and B are aligned, or nearly 
aligned, with direct paths, as illustrated by lines 90, 92, between the cameras and 
the image. Figure 3B illustrates the same ground combat vehicle 12, but instead of 
two pan/tilt cameras as in Figure 3A, the vehicle is equipped with fixed video 

30 cameras 20. In thi s proforrod such embodiments there are eight such fixed 



6 



Replacement Specification (Marked-tip Version) 



Attorney Dkt No. (New) 7784-000966 
Attorney Dkt No. (Old): 66638/41557 

cameras equally circumferentially spaced about a common axis 94 such that 
collectively they encompass a 360° panoramic view about the vehicle. Thus, the 
camera axes 96 of the cameras 20 point radially from the centraf axis 94 and are 45 
degrees apart. Each of the cameras covers at least a 90° panoramic view centered 
5 about its axis such that adjacent cameras overlap the field of view between them. 
This way the entire 360° field of view is covered even when one camera is disabled. 
Thus, two such cameras a and b, viewing the same image as the cameras A and B 
of Figure 3A, have their camera axes 96 substantially out of alignment with the direct 
paths, as shown by lines 104, 106 between the cameras a and b and the image. 

10 For the ANS to perform properly, it must receive two video streams at its 

inputs 1 10, 112 that are substantially the same as the video streams it would receive 
if the image was being viewed by two pan/tilt cameras as in Figure 3A, However, the 
video signals generated by the fixed cameras 20 and stored in the video frame 
buffers do hot produce the same video stream as two pan/tilt cameras, because the 

15 fixed cameras do not actually point in the direction specified by the pan/tilt 
commands 28 from the ANS. Therefore, in accordance with the disclosure 
invention , the Video Formatting component 34 reformats the images to correct the 
image geometry. 

The functions performed by the Video Formatting component 34 can be 
20 implemented as hardware or software. For example, in various embodiments 
Mo s t preferably , the Video Formatting component 34 is implemented using a 
programmable image processing card which may be a purchased off-the-shelf item. 
As illustrated in Figures 3A and 3B, the software on the image processing card 
transforms the image from a fixed camera's focal plane to the image that would be 
25 seen on a pan/tilt camera's focal plane, assuming that the pan/tilt camera was 

pointed in the direction that the ANS commanded. The camera's fbcat plane may be 
a charged coupfed device or any device that transforms light images into electrical 
signals. An image in a video camera is defined by luminous intensity as a fu notion of 
x and y locations on the CCD, i.e., brightness at each pixel. This may be described 
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as i=f{x,y). To correctly transform the image from a fixed camera to make it appear 
like an image from a pan/tilt camera, the software performs certain steps. In 
describing there steps, reference is made to Figures 4a-4g which illustrate an image, 
shown as a tree figure, on the focal plane of a camera {fixed or virtual). 

1 ) convert pixel indices (rowfixed, columnfixed) for the fixed camera focal 
plane to physical locations on the fixed camera's focal plane, i.e., (xfixed, yfixed). 

2) convert the image from r=f1 (xfixed, yfixed), i.e. intensity as a function 
of x and y locations on the fixed camera"s focal plane, to intensity as a function of 
Az-EI coordinates relative to the fixed camera's optical axis, i.e. i=f2 (Azfixed, 
Elfixed). This step yields a curved fbcat plane as shown in Figure 4c. 

3) rotate the Az-EI coordinates from step 2 to the appropriate Az-EI 
coordinates relative to the commanded optical axis of the virtual pan/tilt camera, i.e. 
Azvirtual=g1 (Azfixed, Elfixed) and Elvirtual=g2(Azfixed, Elfixed). 

4) convert the resulting Az-EI image, i=f3{Azvirtual, Elvirtual), to intensity 
as a function of x and y on the virtual pan/tilt camera's focal plane, i.e., I=f4(xvirtual, 
yvirtual). The result of this step is a four-sided polygon that is not ah exact 
parallelogram as it has been transformed to a distorted form from the initial 
paraflelogram of Figures 4a and 4b. 

5) convert the physical locations (xvirtual, yvirtual) to pixel indices, 
yielding i=f5 (rowvirtual, columnvirtual). 

6) Interpolate the pixel values. In general, the center of a pixel in the fixed 
camera will not map directly to the center of a pixel in the virtual pan/tilt camera, so 
the (rowvirtual, columnvirtual) values computed in step 5 will be floating point 
numbers, not integers. Therefore, the Video Formatting component interpolates the 
intensity values given by I=f5 (rowvirtual, columnvirtual), where the values of 
(rowvirtual, columnvirtual) do not align with the centers of pixels, to estimate the 
intensity values at the center of each pixel. 

Steps 5 and 6 yield a focal plane shape that is the same as the shape from 
step 4. Step 5 yields a set of rows and columns that are wider and thicker at the 
front of the view and the rows and columns line up with the edges of the shape. 
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Step 6 yields a set of rows and columns that have uniform width, such that they do 
not line up With all edges of the shape. 

The algorithms used to solve each of these steps are within the skill of those 
skilled in the arts of optics and image processing. 
5 The stitched and reformatted video signals from the Video Formatting 

component 34 are input as two video streams 110, 1 12 to the ANS 26 which outputs 
the pan/tilt commands 28 as previously described, which, if pan/tilt cameras were 
present, would control the direction of the pan/tilt cameras. However, in accordance 
with the present disclosure i nv e ntion , the pan/tilt commands rather than controlling 
10 pan/tilt cameras are input to the Translate to Az-EI component 30 which converts the 
pan/tilt commands into the azimuth and elevation that each pan/tilt camera would 
view if it received those commands. As previously explained, the azimuth and 
elevation values from the Translate to Az-EI component 30 are input to the Az-EI to 
Address Translation component 32 which performs the dual functions of mapping the 
15 azimuth and elevation signals to addresses in each frame buffer 24, and determining 
which frame buffer of the frame buffers 24 to use for each section of the image. The 
algorithms for performing these functions are known in the art. 

The azimuth and elevation values from the translate to Az-EI component 30 
also are input to the Video Formatting 34 which uses these values to determine the 
20 amount of distortion, i.e., the off-axis angle, and which camera to use. 

The display system section 18 comprises an Image Stitcher 120, a Panoramic 
View Buffer 122, an Az-EI to Address Translation component 124, an Az-EI View 
Selectors) 126, and image displays 128. The display system section 18 is known in 
the art for processing video data to display selected images on displays in manned 
25 vehicles for viewing by the crew members. The image stitcher 1 20 receives video 
data from the video frame buffers 124, performs various transformations on selected 
subsets of the video data, and sends the resultant subframe video data to the 
Panoramic View Buffer 122. The buffer 122 contains image data representing a 
360° wide, by approximately 135° high, external view from the vehicle 12. By the Az- 
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EI View Selectors) 126, a crew member selects an azimuth and elevation for 
viewing, and the Az-EI View Selectors) 126 outputs azimuth/elevation data, in 
accordance with those selections, to the Az-EI to Address Translation component 
124, which converts the azimuth/elevation data to a range of memory addresses in 
5 the Panoramic View Buffer 122. The image data from the range of memory 

addresses are output from the Panoramic View Buffer 122 to the displays 128 via the 
Az-EI to Address Translation component 124 to display the selected portion of the 
panoramic view. 

Video from two different cameras with overlapping fields of view is used to 

10 simulate the view of stereo cameras. Selecting different ranges of addresses of 
each of the video frame buffers 24 simulates the effect of panning left or right and 
tilting up or down as with pan/tilt cameras. 

Some ANS systems include a status interface that allows the ANS to check 
the functionality and position of each pan/tilt camera and mount. For such systems, 

1 5 if the pan/tilt cameras were not present, the ANS would interpret the absence of 

pan/tilt cameras as a malfunction and would not operate properly. Thus, the present 
disclosure i nvontion includes a simulator that simulates the status of the pan/tilt 
cameras and mounts. The simulator receives status inquiries from the ANS and 
returns appropriate data. For example, if queried for the current Az-EI of the pan/tilt 

20 camera, the simulator returns the last commanded Az-EI (because that is what a real 
pan/tilt camera would do if operating properly). If queried for operational status, it 
returns an "on" if the fixed cameras are turns on, and an "off* if they are turned off. 

Thus, the virtual pan/tilt camera system and method of the present 
disclosure invent i on , particularly adapted for use with MGVs, e li m i nate s 

25 problomo associated with can replace the use of pan/tilt cameras, i nclud i ng the 
increased decrease vehicle weight and power needs for s uch cameras and the 
difficulty w i th hid i ng s uch camora o reduce risks of detection from radar and 
infrared sensors thereby that can compromise compromis i ng the low-observable 
characteristics needed for military vehicles. The present disclosure invontion 

30 eli minates tho nood for pan/ti l t cameras and thoir associated problem s; 
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providing further provides a system and method that utilizes uti li z i ng fixed 
cameras with known ANS technology. 

The ANS used with the present disclosure invention performs stereo 
imaging processing and autonomously navigates the ground vehicle in the same way 
it performs these functions when used in a pan/tilt camera system. The ANS, when 
used with pan/tilt cameras, receives two video data streams from a pair of pan/tilt 
cameras for the ANS to use to control the cameras by performing stereo image 
analyses. In accordance with the present disclosure i nvention , the same ANS 
receives two video data streams that "appear to" come from two pan/tilt cameras, 
when in fact they are video data streams that have been reformatted from fixed 
cameras. 

While the present disclosure invent i on has been described by reference to 
specific embodiments and specific uses, it should be understood that other 
configurations and arrangements could be constructed, and different uses could be 
made, without departing from the scope of the disclosure i nvention as set forth in 
the following claims. 
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Abstract of the Disclosure 

The present disclosure i nv e ntion is directed to a virtual pan/tilt camera 
system and method for use with vehicles, and particularly ground vehicles (MGVs). 
The disclosure inv e nt i on utilizes autonomous navigation systems (ANSs) used 
wfth pan/tilt cameras, but eliminates the pan/tilt cameras substituting a plurality of 
video cameras. Each video camera is mounted in a fixed orientation on the vehicle 
and covers a selected angular range of lateral view. Commands from the ANS are 
mapped to selected addresses where the video data from the video cameras are 
stored, and appropriately transformed data from the selected addresses are input to 
the ANS. Computers and software in the MGV receive video data from the cameras 
and stitch the imagery together into a single panoramic view. Video data from 
cameras with overlapping fiefds of view are used to simulate the view of stereo 
cameras. 
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